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Quadratic  Programming 

A  Variant  of  the  Wolfe-Markowitz  Algorithms 

Although  a  convex  quadratic  objective  can  be  treated  by  general  convex 
programming,  and  also  can  be  reduced  to  the  convex  separable  case  by  a  change 
cf  variables,  the  linear  nature  of  its  partial  derivatives  has  given  rise  to 
an  elegant  theory  important  in  its  own  light.  It  is  doubtful  at  this  writing 
that  full  potentiality  of  this  theory  has  been  realized. 

Barankin  and  Dorfman  (1958)  first  pointed  cut  that  if  the  linear 
Lagrangian  conditions  of  optimality  were  combined  with  those  of  the  original 
system,  the  optimum  solution  was  a  basic  solution  in  the  enlarged  system  with 
the  property  that  only  one  of  certain  pairs  of  variables  were  in  the  basic  set. 
Markowitz  (1956),  on  the  other  hand,  showed  that  it  was  possible  to  modify  the 
enlarged  system  and  then  parametrically  generate  a  class  of  basic  solutions 
with  the  above  special  property  which  converges  to  the  optimum  in  a  finite 
number  of  iterations.  Finally,  Wolfe  (1959)  proved,  in  an  elegant  way,  that 
an  easy  way  to  do  this  is  to  modify  the  simplex  algorithm  so  as  not  to  allow 
a  variable  to  enter  the  basic  set  if  its  ''congslementary"  variable  is  already 
in  the  basic  set.  Thus  by  modifying  a  few  instructions  in  a  sir^jlex  code  for 
linear  programs  it  was  possible  to  solve  a  convex  quadratic  program.'  We  shall 
present  here  a  variant  of  Wolfe's  procedure.  The  chief  difference  is  a  tighter 
selection  rule  that  results  in  a  monotonically  decreasing  objective  instead  of 
a  decreasing  measure  of  "dual"  infeasibility.  It  is  believed  to  be  computa¬ 
tionally  more  efficient  because  there  can  be  a  greater  decrease  in  the  value 
of  the  quadratic  function  in  each  iteration. 

*  The  name  and  bracketed  date  refers  to  references  at  the  end  of  the  report. 

Other  references  on  quadratic  programs  are  listed  there  also. 


Quadratic  programs  can  arise  in  several  vays.  Wolfe  lists  four  in  his 


paper  as  follows ; 

Regression:  To  find  the  best  least-square  fit  to  given  data,  where  certain 
parameters  are  known  a  priori  to  satisfy  inequality  constraints. 

Efficient  Production:  Maximization  of  profiu,  assuming  linear  production 

functions  and  linearly  varying  marginal,  costs;  see  Dorfman  (l95l)* 

Minimum  Variance :  To  find  the  solution  of  a  linear  program  with  variable 

cost  coefficients  which  will  have  given  expected  costs  and  minimum 
variance;  see  Markowitz  (1959)* 

Convex  Programming:  To  find  the  minimum  of  a  general  convex  function  under 
linear  constraints  and  quadratic  approximation;  see  White,  Johnson 
and  Dantzig  (I958). 


PRELIMINARIES: 

Before  stating  the  problem,  let  us  note  that  every  quadratic  form  can  be 
conveniently  expressed  in  terms  of  a  symmetric  matrix  associated  with  its 
coefficients.  For  example,  for  n  =  5  variables. 


(1)  Q(x)  —  ^  ^22^2  *”55^5  ^  ^*”25^2^5  ^^15^1^5 


^11  ^^12 

^1 

c  c  _  c__ 

12  22  25 

2 

'"IJ  ‘'25  ‘'55 

y_ 

T 

x-'Cx 


where  T  stands  for  transpose. 

T 

Definition:  A  quadratic  form  is  called  positive  definite  if  x  Cx  >  0 

T 

for  all  X  +  0;  it  is  called  positive  semi-definite  if  x  Cx  0  for  all  x. 
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PROBLEM: 

;  Find  x  = 

(x^,  x^,  x^)  0 

Min  Q(x)  satisfying 

(2) 

Ax 

= 

A  .  , 

i  =  1, 

2  ^  ^  m 

T 

X  Cx 

=  Q(x)  , 

C  =  [ey]  , 

1— 1 

It 

2  ^  ^  n 

where 

Q(x) 

is  positive 

* 

semi -definite. 

**  ‘bh. 

Kuhn-Tucker  Optimality  Conditions:  Let  A.,  C  denote  the  j  colixmn  of 

J  J 

A  and  C  and  let 


(5)  y.  .  C^x  -  , 


(tT  --  TTj^,  TT^)  •  •  •  >  TTjj)  • 


THEOREM  1;  A  solution  x  =  x°  is  minimal  if  there  exists  a  TT  =  7T°  , 
y  =  y°  such  that,  for  j  =  1,  2,  n  , 

(4)  Ax°  =  b,  x°  >  (Primal  feasibility), 


(5)  y?  =  cTx°  -  7T°A.  "^0  ,  •  ("Dual"  feasibility), 

J  J  J 

(6)  y?  •  x°  =  0  (Complementarity)  . 

J  J 


PROOF;  Rewrite  Q(x)  in  the  form 

n 

(T)  Q(x)  -  Q(x°)  =  2  ^  (CjX°)(xj-xp  +  (x  -  x°)'^  C(x  -  x°)  . 

J=1 

In  general,  let  x  and  x°  be  any  solutions  satisfying  Ax  =  b,  then 


(8) 


A(x 


n 

-  I  '  °  • 


j=l 


*  ^f  ^^s^red  the  theory  is  easily  extended  to  include  the  addition  of  linear  terms 

**  Theorem  1  is,  as  indicated  earlier,  well  known;  we  reprove  it  because  it 
sets  the  stage  for  the  development  that  follows. 
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Multiplying  on  the  left  by  27r°  and  subtracting  from  (7)  yields,  for  any 
Ax  =  b,  Ax°  =  b, 


(9) 


Q(x)  - 


Q(x°) 


=  2 


n 

E 


(CjX°  -  7r°Aj)(xj-xp  +  (x  -  x°)^  C(x  -  x°) 


where  y°  is  defined  by  (5)  for  x  =  x°. 
J 

x°  •  y°  =  0,  then  (9)  sin^Jlifies  to 
J  J 


If  in  addition  complementarity  holds, 


(10) 


Q(x)  - 


Q(x°) 


=  2 


E 


o 


+  (X  - 


C(x  -  x°) 


y°+0 


Finally,  if  primal  and  dual  feasibility  holds  so  that  x°  ^  0,  x 

J  J 

then  all  terms  in  (lO)  are  non-negative,  therefore  q(x)  ;^Q(x°). 


Improving  a  Non-Optiimal  Solution.  Consider  the  system 

(n)  Ax  =  t  ,  X  0  , 

Cx  -  A*^  7T^  -  y  =  0  ,  (l^:  Identity  Matrix), 

where  x^Cx  is  assumed  to  be  positive  semi -definite .  Let  x°,  7r°,  y°  be  a  basic 

feasible  solution  associated  with  a  basic  set  with  the  complementarity  property; 

namely,  for  each  j  either  Xj  or  y^,  but  not  both,  are  in  the  basic  set.  We 

shall  assume  further  that  the  right  hand  side  has  been  perturbed  to  insure  that 

all  basic  solutions  are  nondegenerate .  Note  that  neither  TX  nor  y  are  sign 

restricted;  only  x  ;^0  is  required  for  a  "feasible"  solution  to  (ll);  eui  optimal 

solution  will  have  been  obtained  if  y.  VO  and  x.  •  y.  =  0  holds  for  all  j. 

J  J  U 
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THEOREM  2:  If  a  basis  Is  ccanplementary  and  y°  <  0,  then  any  increase  of  the 

S  ~  y  ■  ■  .1  ■  ■  .  I 

nonbaslc  variable  vlth  adjustment  of  the  basic  variables ,  generates  a  class  of 

fp 

solutions  ,  y',  such  that  x  Cx  decreases  as  long  as  y'  <  0. 

s 

PROOF:  Let  x  be  any  solution  in  the  class  generated  by  x  and  let  x'be 

generated  by  x^  =  x^.  From  (9),  Q(x)  -  C(x  -  xO 

since  for  all  j  ^  s  either  x^  or  y^  =  0,  The  adjusted  values  of  the  basic  variables 

are  linear  functions  of  x^,  hence  it  follows  that  x  -  x'=  (x  -x')v  where  v  is  a 

constant  vector.  Hence,  Q(x)  -  Q(xO  =  (x^-xp  [^2y'+(Xg-x' )(vV)J  and  it  is  clear 

that  if  y'  <  0,  the  right  hand  side  is  negative  for  sufficiently  small  (x  -x')  >0. 

Moreover  for  Q(x)  to  decrease  with  an  increase  of  x^  ^  0  from  x'  to  x'-',  it  must  be 

accon^janied  by  y'<  y'^ because  Q(x^'')  -  Q(x')=  2(x'"  -  x')y^  +  (x"  -  xM^v^Cv  = 

SS  SSS'SS^ 

""  ■  ^^x  ■  whence  2(y^/  "  “  x^v^^Cv  ^0.  But  v^Cvi:0 

T 

because  v  Cv  =  0  implies  for  positive  semi-definite  forms  Cv  =  0  and  QCx'^-  q(xO  = 

=  2(x''  -  XgOx'Cv  +  (x''  -  xM^vCv  =  0  whereas  Q(x'0  -  Q(x')  <  0;  hence  y"  )>  y'. 

^  s  s 

THEOREM  3:  If  x^  drops  as  basic  variable,  introduction  of  y^  either  causes 

x^Cx  to  decrease  (and  x^  or  y^  to  be  dropped)  or  causes  x^Cx  to  stay  fixed  and 

y^  to  be  dropped.  If  x^  is  dropped,  this  theorem  may  be  reapplied;  on  the  other 

^ud,  if  y^  drops,  either  Initially  or  upon  Increase  of  y^,  Theorem  2  may  be  re¬ 
applied. 


PROOF:  Our  proof  is  completely  general;  however,  for  convenience  we  will 
illustrate  it  on  system  (13)  below.  Let  us  suppose  we  had  on  some  cycle  a  basis 
B  and  a  basic  feasible  complementary  solution  with  basic  variables  x^^,  x^,  Xy 
^4^  ^1'  ^2'  ^5  value  of  y^  =  y°  <  0,  In  this  case,  x^  becomes  a  new 

be.sic  variable  and  we  assume  that  x^  dropped  out  to  form  a  new  basts  B^.  In  (I3), 
the  dot  •  indicates  a  column  in  the  basis  B  and  *  indicates  that  the  column  P^ 
associated  with  x^  is  a  candidate  to  replace  a  vector  of  the  basis  B.  Let  the 
representation  of  P^  in  terms  of  the  columns  of  the  basis  B  be: 


(12) 


Vl  ^  *  ftfe  *  Vt  *  ’’5“5  '  '’5 
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where  is  the  column  in  (ij). 


W) 

^2 

"3 

^4 

"5 

’^l 

^2 

^2  ^3  ^4  ^5 

Const. 

^11 

^12 

^13 

^4 

^15 

i 

agi 

^22 

^23 

^24 

^25 

i 

i 

^2 

"n 

=12 

''13 

=14 

'^15 

\l 

api 

-1 

0 

=22 

^23 

=24 

=25 

\2 

^22 

-1 

0 

''13 

""23 

''33 

''34 

=35 

^15 

^23 

-1 

0 

^14 

=24 

''34 

=44 

=45 

^14 

^24 

-1 

0 

^^25 

''33 

""43 

=55 

^15 

^25 

-1 

0 

• 

• 

• 

• 

* 

• 

• 

• 

basis  B 

• 

• 

• 

• 

• 

• 

* 

basts  B ' 

Let  us  now  consider  the  representation  of  the  column,  in  terms  of 

the  basis  B*  where  are  the  weights  on  columns  associated  with  basic  x^, 
TT^  and  5^^  are  the  weights  on  columns  P^  associated  with  basic  y^  , 


(lita) 


■■A  *  ^2^  *  '’sS  *  "’sS  ^  ^b>i  *  "’t’v  *  '■5^  ' 


We  wish  to  show  that  ^  0.  If  <  0,  it  is  clear  that  an  increase  of  y^^ 

T 

will  cause  to  increase  and  x  Cx  to  decrease  as  long  as  the  value  of  y^  <  0 

in  the  basic  solution.  On  the  other  hand,  if  7^^  =  0,  we  shall  show  that  y^ 

T 

will  drop  with  no  change  in  x  Cx. 


(i.e. 


Let  [?v^]  be  the  representation  of  Pj^  in  terms  of  the  prior  basts  B, 
,  before  the  introduction  of  x^  in  place  of  Xj^), 


(l4b) 


^l\ 


P2>2 


.P3.5 


+  P.^\^ 


+  P, 


5^ 


=  P, 
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Then,  setting  A  =  (?v^,  A^,  Ay  A|^),  the  first  six  rows  of  this  representation 
yields  (15 )  and  (l6) 


(15) 

[an 

^12 

"13 

hi,’ 

a" 

= 

0 

^^21 

^22 

^25 

hi.’ 

A^ 

= 

0 

(16) 

“n 

"12 

""13 

^14 

^■11 

“  0” 

^12 

C22 

""23 

^24 

A% 

^12 

h  + 

^22 

0 

"13 

""25 

""33 

""34 

"13 

^25 

0 

^11 

^24 

^3^ 

^44 

^4 

^24 

-1 

Multiplying  (16)  by  A  on  the  left  and  denoting  the  square  matrix  by  yields, 
T  T 

(15  "^l4*  Since  AC^^^A  is  positive  semi -definite  (C^  is  a  principal 

minor  of  C),  ACj^A^  0  and  0  follows. 


Case  A[^  <  0:  Let  us  assume  A|^  <  0.  We  observe  that  in  the  representation 
(12)  of  in  terras  of  B,  the  weight  is  positive  (since  decreased  when 
increased).  By  elimimting  from  (12)  and  (llo)  to  obtain  (l4a),  and  noting 
aj^  >0,  A^^  <  0,  it  follows  that  A^  =  <  0  (where  ^  is  the  weight  on  P^  in 

the  representation  of  Pj^  in  terms  of  B^).  But  A^  <  0  Lmp.Mes  thoat  the  introduc¬ 
tion  of  y^^  into  the  basic  set  for  B*  will  increase  x^.  Moreover,  we  may  adopt 

the  point  of  view,  for  the  purpose  of  the  proof,  triat  it.  is  the  increase  in  x^ 

5 

that  is  "causing"  the  increase  in  y^^  (instead  of  the  other  way  around),  so  that 
we  are,  in  fact,  repeating  the  situation  just  cor.si(i.ered  of  increasing  x^  and 
adjusting  the  other  "basic”  variables,  except  here  y^^  is  in  the  basic  set  in¬ 
stead  of  X, .  It  follows,  therefore,  as  before,  that  an  increase  in  x  decreases 
t  5 

T 

X  Cx  as  long  as  y^  remains  negative  in  val.ue  in  the  adjustment  of  the  basic 
solution  by  the  increase  of  x  . 


-7- 


Case  =  0:  Let  us  now  assume  X,  =  0.  We  may  set  X.  =  V.  because  the 

-  5  4  1  i 

representation  of  is  the  same,  whether  in  terms  of  B  or  B* ;  hence,  ?^  =  0. 

T  I  T 

In  this  case  =  -A^=  0;  therefore,  =  0  by  a  well  known  property  of 

semi -definite  forms.  In  this  case  X  =  0  must  hold  because  X  4  0  implies  a 
dependence  of  the  first  four  columns  of  (15)  and  (l6)  which  is  impossible  be¬ 
cause  then  the  square  array  of  coefficients  of  (15)  and  (I6),  and  in  turn  B, 
would  be  singular. 


Setting  X  =  0  in  (16)  and  noting  that  at  least  one  X^  must  be  nonzero, i=l,.. ., 7, 


X  associated  with  the  basic  set,  other  than  x^.  By  forming  a  linear  combina- 

J  ~  '  1? 


tion  of  the  rows  of  A,  we  could  therefore  rewrite  (for  the  purpose  of  the 


proof)  the  ; 

(17) 

system 

so  that 

top  row  has 

zero 

coefficients  for  these  x.. 

J 

Thus 

^1 

^2 

"3 

^4 

x_ 

^1 

^2 

^1  ^2  ^3  ^4  ^5 

Const. 

0 

0 

0 

^14 

“15 

/ 

"1 

^21 

^22 

"25 

^24 

^25 

"2 

"u 

"12 

^^14 

^^15 

0 

^21 

-1 

0 

^^12 

^^22 

^^24 

^=25 

0 

^22 

-1 

0 

"35 

0 

1 

"23 

-1 

0 

'^14 

^"24 

C44 

^4 

^24 

i 

-1 

0 

''15 

^=25 

^^45 

'=55 

^25 

-1 

0 

• 

• 

• 

• 

1 

• 

* 
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A'cw,  a'  ^  J:  0  because  B  was  nonsingular  and  a/^  0  because  %y.xe  sa:m  is 

zrue  for  B'’  ■  It  is  also  obvious  tliat  the  signs  of  ^  and  a.|^  are  the  saiae 

for  X,  to  decrease  when  x..  increases.  Note  now  that  the  'in'  column  is  reprc- 
••i-  t  U- 

senrable  as  s.  linear  combination  of  the  negative  unit  columns  of  ,  y  (and, 
in  a  'tore  general  case  than  the  example,  the  other  negative  unit  ec'lumns  of 


;ne 


basic  Y  ).  Moreover  it  is  clear  that  since  a!,,  and  a(,_  have  the  same  sign, 
'  L  ^4-  .i. 


ir.cre.'.siiiii  /,  fx'om  its  sero  varue  results  in  a  positive  change  In  y. . 

•4  J 

Since  tne  y;  are  n')o  sign  restricted,  y|^  can  be  increased  util  y_  is 

oroopeu.  o\.t  I'd’  the  basic  set  c-t  value  zero  becax-se  ah.,  x,  ’v;:'..tu',;s  p.re  unaffected. 

'I' 

Hence,  .In  t-r.s  snift  of  basis  tnere  is  no  chxnge  in  she  val-re  of  z'^Cx;  however, 

the  i..'iU.;'odu -•nio.'i  of  y,  into  t.ie  basic  set  and  dropping  of  y  ,  gives  rise  to  new 

4 

basic  set  tiian  satisfies  the  comp.-2me.ntHrity  p.vopc;rty.  he  rmy  thus  o.pplj;  again 
l^moo'cr:  t.)  red.\ce  x-Ca. 


QHuKAl!.lC  .iij-l'jhil'liyi 
flDil'  i 


Ln  - 1..  1- ce ;  Ixi'c  Ax  --  b  be  a  basic  xcr 


;  ;  c.'ie  f  >i  tue  iuiu,;.!  sot  cjf  hc.'i.l  ?  \ i.,;i.'..'..  .  .i';.:-  u  k- 

fnilorgf.'d  pi’'.'.rle-...i.  (  xj  oi.'esx  >:  ,  (bj  o‘-(.’  ipiccv. le. .us,  y  , 


• /..  ll  'v  ^  h».  xt’.  -/  »l.  0 

<j  *  * 


'lOt,  ■.  r  c  jc::  I  icieni 


y.  l.jr  the  vtiLticS,  d 
u 


o 

''  i 


tli’..  SaS„C  S  .'.‘.'..’bj.  CL' ,  '  C  :  .  .T'.t- i'C 

i'l'.n  y .  ■■  y'h.  If  y  '  >  0  tcmrJ.nate;  th.^  solu.cio.^,  is  it;;.  !: i .ail „ 
If  y^  <  0  inti’oduce  into  b.isic  set  x,,;  i-f  y,  drops  .f.;am 

ki  '  Ok 

basic  set,  repeat  huepi  2  Otheiwlse  if  .x„.  drops, 

h'l't'lP  ;h  Tixti’.jduce  into  basic  set.  If  j.,  Imps,  I'aturn  to  hr.cp  g; 

'inherrise,  if  some  x^,  drops,  repeat  Step  5  with  playing 

Z  -f  -Zr 


.rie  role  of 


THEOREM  4.  The  iterative  process  ir>  t’inihe. 

PROOF:  The  niimter  of  possible  basic  sets  is  finite.  Each  one  generated  by 
the  process  is  different  because  of  the  decreases  in  xCx.  But 
this  means  the  cyclic  process  must  terminate. 


O- 


CONCLUSION: 


Formula  (lO)  is  the  analog  for  quadratic  programs  of  the  familiar 

adjusted  objective  function  obtained  by  elimination  of  the  basic  variables 

in  linear  programming.  [For  general  convex  objectives,  it  appears  to  be  a 

natural  take-off  for  a  quadratic  fit.]  If  the  coefficients  y°  of  the  x.  are 

J  J 

non-negative,  the  solution  is  optimal.  If  not,  a  new  basic  solution  for 

system  (ll)  is  obtained  by  increasing  x  corresponding  to  y°  =  Min  y°. 

s  s  j 

Either  y  drops  out  as  basic  variable  or  y  drops  after  a  sequence  of  replace- 
s  s 

ments  of  basic  variables  x^  by  their  correspondents  y^.  With  the  latter  pro¬ 
vision  for  a  decrease  in  the  dimensionality  of  the  solution,  the  algorithm 
may  be  viewed  as  a  direct  extension  of  the  regular  simplex  method  to  quadratic 
programs  (in  contrast,  the  algorithms  of  Wolfe  and  Markowitz  may  be  viewed  as 
parametric  extensions). 
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